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ABSTRACT: Computational theories of structurc-from-motion and stereo vision only specify 
the computation of three-dimensional surface information at points in the image at which tire ir- 
radiancc changes — for example, the zero-crossings of a V 2 G operator applied to the image. Yet, the 
visual perception is clearly of complete surfaces, and this perception is consistent for different observ¬ 
ers. Since mathematically the class of surfaces which could pass through the known boundary points 


provided hv the stereo svstem is infinite and contains widely varying surfaces, the visual system must 
incorporate some additional constraints besides the known points in order to compute the complete 
surface. 


Using die image irradiance equation, we derive the surface consistency constraint, informally 
referred to as no news is good news. The constraint implies diat the surface must agree with the 
information from stereo or motion correspondence, and not vary radically between diesc points. An 
explicit form of this surface consistency constraint is derived, by relating the probability of a zero¬ 
crossing in a region of the image to die variation in the local surface orientation of die surface, 
provided diat the surface albedo and the illumination arc roughly constant. 


The surface consistency constraint can be used to derive an algorithm for reconstructing the 
surface that “best” fits die surface information provided by stereo or motion correspondence. 
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1. Introduction 


Although our world has three spatial dimensions, the projection of light rays onto the retina 
presents our visual system with an image of the world that is inherently two-dimensional. We must 


use such images to physically interact with this three-dimensional world, even in situations new to 


us, or with objects unknown to us. That we easily do so implies that one of the functions of the 
human visual system is to reconstruct a three-dimensional representation of the world from its two- 
dimensional projection onto our eyes. 


Methods that could be used to effect this three-dimensional reconstruction include stereo vision 


[c.g. Wheatstone, 1838; Helmholtz, 1925; Julcsz, 1971] and structure from motion [for example, Miles, 
1931; Wallach and O’Connell, 1953; Johannson, 1964], Both of these methods may be considered as 
correspondence techniques, since they rely on establishing a correspondence between identical items 
in different images, and using die difference in projection of these items to determine surface shape. 

Most of the current computational theories of these processes [Marr and Poggio, 1979; Grimson, 

1980, 1981a; Mayhew and Frisby, 1981; Ullman, 1979; Longuct-Higgins and Prazdny, 1980] argue 

that the correspondence process cannot take place at all points in an image. Rather, the first stage 

of the correspondence process is to derive a symbolic description of points in the image at which the 

« 

irradiance undergoes a significant change [Marr and Hildreth, 1980]. This symbolic representation 
(called the primal sketch [Marr, 1976; Marr and Hildreth, 1980]) forms the input to the second stage 
of the process in which tine actual correspondence is computed. As a consequence of die form of the 
input, the correspondence process can compute explicit surface information only at scattered points 
in the image. Yet our perception is clearly of complete surfaces. For example, in Figure 1, a sparse 
random dot stereogram yields die vivid perception of a square floating in space above a background 
plane, rather than a collection of dots suspended in space. This suggests that some type of filling-in, or 
interpolation of surface information, is taking place in the visual system. 


This poses an interesting problem, since mathematically the class of surfaces that could pass 
dirough die known boundary points provided by die stereo algoridim is infinite and contains widely 
varying surfaces. The implication of our perception of complete surfaces, which is consistent for 
different viewers, is that die visual system must incorporate some additional constraints, besides the 
known points, in order to compute the complete surface. 
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Figure 1. A Sparse Random Dot Pattern. Although the density of dots is very small, the perception 
obtained upon fusing this pattern is one of two disjoint planes, rather than dots isolated in depth. 


In this paper, wc will use the image irradiance equation [Horn, 1977] to derive the surface 
consistency constraint, informally referred to as no news is good news. The constraint implies that the 
surface must agree with the information from stereo or motion correspondence, and not vary radically 
between these points. Wc will derive an explicit form of this surface consistency constraint, by relating 
die probability of an abrupt irradiance change in a region of the image to tire variation in the local 
surface orientation of the corresponding surface. Wc will then indicate how this constraint can be 
used to derive an algorithm for rcconstaicting the surface that “best" fits the surface information 
provided by stereo or motion correspondence [Grimson, 1981b, 1982], 

2. The Computational Constraint 

The input representation used by tire stereo algorithm [Marr and Poggio, 1979; Grimson, 1981a, 
1981b] consists of a symbolic representation of points at which die image irradiance undergoes a 
significant change. These points are identified by die zero-crossings of a Laplacian of a Gaussian 
(V 2 G) operator applied to the image; that is, the image is convolved with a V 2 C operator and the 
points at which tire resulting convolution changes sign are located [Marr and Hildreth, 1980; Hildreth, 







1980]. As a consequence of the correspondence process, explicit three-dimensional information about 
the surface shape will be computed only at these zero-crossings. 

Suppose one were to attempt to construct a complete surface description based only on the 
surface information known along the /cro-crossings (where by complete, we mean that an explicit 
suiface value is assigned to each point on a rectangular grid). An infinite number of surfaces would 
consistently fit the boundary conditions provided by these surface values. Yet there must be some 
way of deciding which surface, or at least which small family of surfaces, could give rise to the zero- 
crossing descriptions. This means that there must be some additional information available from the 
visual process which, when taken into account, will identify a class of nearly indistinguishable surfaces 
that represent the visible surfaces of a scene. 


In order to determine what information is available from the visual system, one must first care¬ 
fully consider the process by which the zero-crossing contours are generated. For instance, sudden 
changes in the reflectance of a surface, caused by surface scratches or texture markings, will give rise 
to zero-crossings in the convolved image [Marr and Hildreth, 1980; Hildreth, 1980], Of more interest 
uei c is the iact mat sudden or sharp changes in die shape of die surface will under most circumstances 
also give rise to zero-crossings. This can be used to constrain die possible shapes of surfaces diat could 
produce particular surface values along the zero-crossing contours 


We illustrate die basic argument with an example. Suppose one is given a closed zero-crossing 
contour, within which there arc no other zero-crossings. An example would be a circular contour, 
along which die dcpdi is constant. There are many surfaces which could fit this set of boundary 
conditions. One such surface is a flat disk. One could, however, also fit other smooth surfaces to this 
set of boundary conditions. For example, the highly convoluted surface formed by sin(x 2 + y 2 )* 
would be consistent with die known disparity values (sec Figure 2). Yet in principle, such a rapidly 
varying surface should give rise to other zero-crossings. This follows from the observation diat if the 
surface orientation undergoes a periodic variation, dien it is likely that the irradiance values will also 
undergo such a variation. Since the only zero-crossings lie at die borders of the object, diis implies 

diat the surface sin ( x 2 + y 2 ) 2 is a highly improbable representative surface for this set of boundary 
condidons. 

Hence, die hypodiesis, which will be evaluated in die following sections, is diat the set of zero¬ 
crossing contours contains implicit information about the surface as well as explicit information. If 
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Figure 2. Possible Surfaces Fitting Depth Values at Zero-Crossings. Given boundary conditions 

of a circular zero-crossing contour, along which tire depth is constant, there are many possible 

surfaces which could fit the known depth points. Two examples are a flat disk, and tire highly 
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convoluted surface formed by sin (x 2 -j- y 2 y, shown here. (From Grimson [1981b].) 


one can determine a set of conditions on the surface shape that cause inflections in die irradiance 
values, then one may be able to determine a likely surface structure, given a set of boundary condi¬ 
tions along die zero-crossing contours. 

2.1 No News is Good News 

% 

In general, any one of a muldtude of widely varying surfaces could fit die boundary conditions 
imposed along die zero-crossings. Our intention is to show that to be completely consistent with 
die imaging process, such surfaces must meet both explicit conditions and implicit conditions. The 
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explicit conditions arc given by the depth values along the zero-crossing contours. The implicit condi¬ 
tions arc that the surface must not impose any zero-crossing contours other than those which appear 
in the convolved image. This implicit condition leads to the surface consistency constraint [Crimson, 
1981b], namely: 

The absence of zero-crossings constrains the possible surface shapes. 

Just as the presence of a zero-crossing tells us that some physical property is changing at a given 
location, the absence of a zero-crossing tells us the opposite, that no physical property is changing, 
and in particular that the surface topography is not changing in a radical manner. We informally 
refer to this constraint as no news is good news, since it says that in general the surface cannot change 
radically without informing us of this fact by means of zero-crossings. Note that in practice, the 
convolved image may not signal the entire extent of a discontinuity in the surface topography, but at 
least some portions of the discontinuity will be evidenced by a zero-crossing. In order to complete 
such a partial discontinuity, a subjective contour must be inserted between the known portions [Brady 
and Grimson, 1981]. 

In order to make explicit any constraints on tire shape of the surface for locations in the image 
not associated with a zero-crossing, we will carefully examine die factors which influence irradiance 
and hence image intensity. (Note drat while irradiance refers to die illuminant flux emitted from the 
surface of an object and intensity refers to die brightness recorded by a sensor, we will use the terms 
interchangably, since they are proportional [Horn and Sjobcrg, 1979].) These factors will be expressed 
in the image irradiance equation, which describes die manner in which a particular irradiance is 
formed at a point in the image. Two goals are kept in mind. The first is to determine what surface 
conditions will cause a local change in irradiance, and the second is to combine this constraint with 
die input from the visual processes, such as stereo or structure-from-motion, in order to design an 
algoridim for interpolating surface information. 

The basic result, corresponding to the intuitive argument given above, is diat the probability of a 
zero-crossing occurring, in regions where the illumination is roughly constant and die surface material 
does not change, is a monotonic function of die variation in the orientation of the surface normal. 
This means diat the probability of a zero-crossing increases as the variation in surface orientation 
increases. By inverting this argument, we will show diat die best surface to fit die known data is that 
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which minimizes the variation in surface orientation 


since this surface is most consistent with the zero- 


crossings in the convolved image. 


3. Image Formation 

In order to examine the process of zero-crossing formation, a review of the factors involved in 
die formation of an image will be presented. One factor is the geometry of the projection (cither 
orthographic or perspective) from die scene to the image. The second factor, which we will review 
briefly below, is the manner in which image irradiancc values at a point are formed. A detailed 
analysis of Uicsc processes has been undertaken by several investigators [Horn, 1970, 1975, 1977; 
Nicodcmus, ct al., 1977; Woodham, 1978; Horn andSjobcrg, 1979]. 


3.1 Grey-Level Formation 


In this section, we will outline the factors involved in the creation of image irradianccs in order 
to illustrate the relationship between changes in irradiance and zero-crossings in die convolved image, 
(most of the discussion is based on Horn and Sjoberg, [1979]). To do this, we need to determine what 
irradiancc value will be associated with a particular image location. 

The apparent “brightness” recorded by an imaging device is a measurement of image irradiance 
E, the radiant flux striking a unit area of the receptive field. The flux reaching a small portion of 
the receptive field will be exclusively a function of a corresponding small surface clement on some 


object in the scene, provided die imaging system is properly focused and die lens is small relative to 
its distance from die object [Horn and Sjoberg, 1979]. We want to specify die factors diat determine 
the image irradiance at a point in the image, as a function of die corresponding surface element. 

In general, there are direc factors governing the irradiance recorded at any position in the 
image: 

(1) the amount of radiant flux striking a surface, a function of die distance of die surface from the 
source r, the orientation of the surface relative to the source, and the intensity I of the source 
itself; 

(2) the percentage of incident flux reflected by die surface (as opposed to the percentage absorbed 
or transmitted), a function of the surface material and usually described by die albedo p ; and 





(3) the distribution of that reflected flux as a function of direction, usually described by a reflectance 
function. 


We shall show that a sharp change in each of these factors will give rise to a zero-crossing in die con¬ 
volved image. Further, in regions where both die incident illumination and absorption characteristics 
of the surface material arc roughly constant, an equation relating die recorded irradianccs to the shape 
of die surface will be derived. This image irradiancc equation will be used in later sections to verify 
our surface consistency constraint. 

Most surfaces have die property that the reflectance is not changed by rotating a surface element 
about its normal (exceptions arc diffraction gratings, irridcsccnt plumage and “tiger’s eye”). Such 
surfaces are referred to as isotropic. If a surface possesses this property, only diree angles are needed 
to determine reflectance, as shown in Figure 3. The angle between the local surface normal and the 


incident ray is called the incident angle and is denoted by i. The angle between die local surface 
normal and the emitted ray is called the view angle and is denoted by e. The angle between the 
incident and emitted rays is called the phase angle and is denoted by g. In the following, we shall 
assume that the surfaces arc isotropic. This reduces die image irradiancc to a function of/, r, i,e, g 
and p, where I is the intensity of the source, r is the distance of die source from the surface, i, e, and g 
are die angles specified above, and p is die albedo of die surface material. 

We know that the incident flux density follows an inverse-square reduction as a function of dis¬ 
tance. If two surfaces lie at very different depdis, and all other factors are roughly equal, diis inverse- 
square dependence will cause a noticeable difference in the irradianccs associated with the different 
surfaces. If die projections of the two surfaces are adjacent in the image, die different irradiances 
will give rise to a zero-crossing in die convolved image. This is to be expected, since the Marr- 
Hildreth theory of edge detection was based on the requirement of detecting changes in the image 
corresponding to changes in some physical property of the surfaces. In this case, the changing physical 
property is object continuity. 

Given diat these situations will cause a zero-crossing, we can restrict our attention to regions of 

the image between the borders of the objects. We will further assume that die source is distant relative 

* 

to die surfaces. This has two consequences. The first is that each surface receives roughly uniform 
illumination. This implies that the 7/r 2 term is roughly constant and may be ignored in what follows. 
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Figure 3. Hie Imaging Geometry. The incident angle i is the angle between the incident ray 
and surface normal. The view angle e is the angle between the emergent ray and the surface 
normal. The phase angle g is the angle between the incident and emergent rays. (Redrawn from 
Woodham, [1978]). 


The second is that the angle between the viewer direction and the source direction (the phase angle g) 
will be roughly constant over each surface. ' 

If the surface material changes, there will frequently be an associated change in the albedo. If 
all other factors are roughly equal, tins will cause a change in tine irradiances and there will be a 
corresponding zero-crossing in the convolved image. This is again expected from the Marr-Hildreth 
theory of edge detection: a changing physical property of the surface should correspond to a zero¬ 
crossing in the convolved image. In general we shall restrict our attention to regions of tire surfaces 
between such material changes, so that the albedo p is roughly constant. In these regions, we can 
consider the image irradiances to be a function of i, e and p (ignoring scale constants). 

If we assume that the source is distant enough and small enough to be treated as a point source, 
then we can simplify die geometry further by reducing the angles i and e to a specification of die local 
surface normal, in the following manner. Consider a Cartesian coordinate system with z -axis aligned 
with the line of sight, and z-axis some arbitrary direction in the plane normal to die line of sight. 
Suppose die surface is specified in die coordinate system by z — f(x, y). Elementary calculus states 








that the vectors 


{*■•>1} {^'I} 


are tangent to the surface at a given point. The cross product 


f dz dz \ 

W<V / 


is normal to the tangent plane, and hence to the surface. Defining 


V 


dz 

dx 


dz 

dy 


the normal vector becomes {p, q, —1}. The quantity (p, q) is usually referred to as the gradient and 
the space of all points (p, q) is referred to as gradient space [Huffman, 1971; Mackworth, 1973]. 

In this case, the angles i and e can be straightforwardly transformed into the normal components 
p and q [Horn, 1977]. Since we are considering the ease of a distant point source, its direction relative 
to the surface can be represented by a direction vector {p„ q H) —1}. A straightforward calculation of 
the vector dot products shows that tire angles i and e arc given by: 


cos(i) 


cos(e) 


1 + PPs + QQs 


\A + P 2 + 9 2 \A + Ps -H Qa 

1 


\/l + P 2 + <7 2 


Extended sources (sources whose size relative to their separation from the scene cannot be treated 
as a point source, for example, fluorescent lights in a normal room) can be frequently treated by su¬ 
perposition of point sources, or as being equivalent to an appropriately chosen point source [Brooks, 
1978; Silver, 1980]. Thus, the irradiance reduces to a function ofp, q and p. 

Finally, if we assume that tire objects are distant relative to the viewer, the image projection may 
be treated as orthographic. In this case, the coordinate system of the image and the coordinate system 
of tlie scene can be treated as identical so that we obtain the image irradiance equation: 

E{x, y) = p{x, y)R( p(x, y), q(x, y)) 

where E(x, y) is the image irradiance recorded at a point in tire image, p is the albedo associated 
with tlie surface intersecting tlie ray from ( x, y) and R{p, q) is tlie reflectance map [Horn, 1970, 1975, 


mm# 
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Horn and Sjobcrg, 1979, Horn and Bachman, 1977]. Note that the reflectance map R differs from 
the reflectance function for a surface [Nicodemus, ct al., 1977], The reflectance function describes 
the amount of flux reflected in a particular direction, given an incident flux from a second given 
direction. The reflectance map combines the effects of the reflectance function in the case of uniform 
illumination and isotropic surface material with the illumination geometry and the image sensor’s 
view point. In the case of constant albedo the image irradiance equation reduces to the partial 
differential equation [Horn, 1970,1975; Horn and Sjobcrg, 1979]: 

E[x, y) = R( p[x, y), q(x, y )). 


These two equations describe the manner in which an image irradiance (or grey-level) is obtained at a 
particular point in the image. 


4. The Interpolation Theorem 

We have seen that zero-crossings can arise from many factors. We shall restrict our attention to 
regions in which illumination is constant, albedo is roughly constant and surface material is isotropic. 
Our intention is to show that in such cases, if the surface topography changes radically, the image 
irradianccs must also change radically. We will then be able to use die contrapositive statement— if 
die image irradianccs do not change radically, dicn die surface topography also does not change. Note 
diat if die albedo or die illumination is not constant over die region, we cannot make this statement. 
That is, there could be situations in which die surface topography does change radically without 
causing a corresponding change in the image irradianccs, because one of die other factors, such as 
the albedo, is also changing in such a manner as to mask out die effects of die change in topography. 
These situations are fortunately very rare, since diey require a precise meshing of die effects of die 
changing albedo with die changing surface topography and in general are highly dependent on the 
viewer position and illuminant direction. Thus, a small change in die viewer position will usually 
decouple die conflicting effects of albedo and surface shape, and die change in topography will give 
rise to a zero-crossing. (In other words, it is generally difficult to paint a radically curved surface so 
as to appear flat from all viewpoints. The opposite is not true, of course. One can easily paint a 
flat surface to appear curved. In other words, we prefer to see topographic changes, as evidenced by 




the informal observation that photographs with smoothly changing albedo are usually interpreted as 
having smoothly changing topography.) 

The concern here is whether there can be zero-crossings corresponding to topographic rather 
than photometric changes in an object, for example, in the radial sine surface of Figure 2. Indeed, 
if the albedo is roughly constant, tire form of the equations indicates that there may be topographic 
effects that could also cause sharp changes in irradiancc. The surface conditions under which such a 
change in irradiance can occur are important, since the absence of zero-crossings in a region would 
then imply the absence of such surface conditions for that region. It is precisely these restrictions 
on surface shape which will allow us to detennine a surface consistent with the depth values along 
the zero-crossing contours. The basic problem is, under what conditions docs bending of the surface 
force an inflection in the irradiancc array? This question will be answered in the following sections by 
considering specific cases. 

There are two points to note, before beginning the mathematical details. The first concerns the 
role of the Gaussian filter in constructing the primal sketch representations upon which the stereo 
correspondence is computed. The Gaussian performs a smoothing of the image, thereby isolating ir¬ 


radiancc changes at a particular scale. At the same time, it removes some of the noise problems which 
arise from using a discrete grid to represent the image. In what follows, we shall concentrate on the 
differential operator (V 2 ) and its role in the creation of the zero-crossings. The effect of die Gaussian 
will not be considered in any detail. Hence, the surface reconstruction will not account for minor 
surface fluctuations on the scale of the grid resolution. This is not a major problem. (Equivalently, 
the problem may be considered as one of applying die Laplacian V 2 to die image G * E. The 
reconstruction will be based on this image information.) 

The second point concerns die use of the Laplacian operator V 2 as opposed to directional 

/ . 

derivatives. The mathematical arguments which follow are based on the consideration of zero- 
crossings of the convolution of a directional second-order differential operator with the image, (v • 
V) * E. We have already seen that the Marr-Hildrcth theory of edge detection is based on the use 
of zero-crossings of the image convolved with the Laplacian, V 2 * E. This difference is not critical, 
for die following two reasons. Marr and Hildreth show diat under some simple assumptions, the 
zero-crossings obtained by eidier operator are identical. In particular, the condition of linear variation 
[Marr and Hildredi, 1980] states that: the irradiance variation near and parallel to a line of zero- 
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crossings should locally be linear . If this condition is assumed to be true then it can be shown [Marr 
and Hildreth, 1980, Appendix A] that the zero-crossings obtained with the Laplacian are identical to 
those obtained using directional derivatives. Thus, in those situations for which the condition of linear 
variation is valid, no difference is obtained. In the case where tine condition of linear variation docs 
not hold, the difference in the zero-crossings of the operators lies in their position, not in the existence 
of such zero-crossings. For example, at a corner (which frequently carries strong visual information), 
both V 2 -I- E and (v * V) * E will give rise to zero-crossings. The only difference will be in the exact 
position of the zero-crossings. We shall see that such variation in position is of negligible consequence 
to the reconstruction process. Hence, it will be assumed that the arguments developed in the next 
section, based on the zero-crossings of a directional second derivative will also generally apply to the 
zero-crossings of the Laplacian. 

Finally, in what follows, the functions p, R and / arc assumed to have continuous second order 
partial derivatives. Throughout, it is assumed that the albedo effects can be ignored relative to the 
topographic ones, that is, the albedo factor may be considered roughly constant and will therefore not 
affect any study of derivatives. Without loss of generality, one may assume that p = 1 and ignore the 
albedo in what follows. 


4.1 General Argument 


The basic hypothesis is that in order for a surface to be consistent with a given set of zero- 

crossings, not only must it give rise to a zero-crossing in the convolved irradiances at those points, 

but it must also not give rise to zero-crossings anywhere else. Under most situations, this restriction 

would require that the surface not change in a radical manner between zero-crossing contours. (For 

example, the surface shown in Figure 2 is not consistent with the boundary conditions of a circular 

zero-crossing contour of uniform depth). It is difficult to prove this assertion in general, since die 

image formation equation includes terms dependent on the imaging process and on the light source 

geometry, as well as factors dependent on the photometric properties of the surface itself. However, 

under some fairly weak assumptions concerning the relative strengths of photometric and topographic 

* 

changes, and the form of the reflectance function, we will prove the surface consistency theorem, 
describing the probability of a zero-crossing as a function of the shape of the surface. The importance 
of this theorem is that it leads to a method for measuring the probability of a particular surface being 





Figure 4. An Example of a Developable Surface. The component of the surface orientation in 
the y direction is constant for this region of the surface, so that the only variations in surface 
/'“A orientation take place in die x direction. 


inconsistent with the zero-crossing information. This in turn suggests that it will be possible to derive 
a method for determining the best possible surface to fit the known information [Grimson, 1982], 


4.2 A One-Dimensional Example 


To illustrate the scope of the surface consistency theorem, we shall consider first the one¬ 
dimensional case of a developable surface, before proving the general theorem concerning arbitrary 
surfaces. Note that the Laplacian is orientation independent, so that without loss of generality, one 
may rotate the coordinate system of the image to suit our needs. One may assume that the surface has 
the form f(x, y ) such that f y (x, y ) = q{x, y) = c, in the local region under consideration. Hence, 


the partial derivatives of q vanish, as do any partial derivatives of p involving y. A sample surface is 



shown in Figure 4. 

Suppose that a one-dimensional slice in the x direction of the surface contains at least two 
inflection points. Figure 5 indicates a sample surface and its derivatives. Since q is assumed constant, 
the derivatives of the image irradiance equation are given by: 
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+ 2 pxRpPx + P^-ppPl + RpPxxS- 

(Note that except where explicitly stated otherwise, we shall use a subscript to denote a partial deriva¬ 
tive, so that, for example, p x = dp/dx.) In tit is situation of a developable surface, we can prove the 
following result. 

Theorem 0: Consider a portion of a second differentiable, developable surface oriented along the y 
axis such that f y {x,y) — q = c, for some constant c. If the following conditions are true: 

1. The surface portion contains exactly two inflection points in the x direction, at x x andx 2 , 

2. At the points X\ and x 2 , normalized changes in albedo are dominated by normalized changes in 
reflectance, 

V 2 p Rpp xx 

~ < ~1T~ ’ 

3. The reflectance map R does not pass through an extremum in this region of the surface, 

4. The reflectance mapR is not constant over this region of the surface, 

5. The albedo p is non-zero, 

then there exists a point x { < x' <_ x 2 such that V 2 E(x > ) — 0. That is, there exists a zero-crossing 
here. 

Proof. The signum function is defined by: 

( 1 if x > 0 
sffn(x) = < —1 if x < 0 
, 0 if x = 0 

From the derivatives of the image equation, 

— (V 2 p)/Z -f- 2 p T Rpp x + p(R pp p 2 x -f- RpPxx)- 

At the inflection points xi and x 2 , p x {xi) — 0. Hence, evaluation of the equation yields 

V 2 E(xi) = V 2 p(xi)R(p(x t )) + p{xi)R p (p{xi))px X {xi) for* = 1,2. 

Condition (2) implies that the albedo changes are negligible in this region, so that the first term may 
be ignored, 

sgn(y 2 E(xi)) = sgn{p(x l )R p {p{x l ))p xx {x l )). 





Figure 5. One-Dimensional Example. The top figure illustrates a slice of the surface, containing 
two inflection points. The second figure illustrates the first derivative of the surface function. The 
two inflection points of the surface correspond to extrema in the first derivative. The third figure 
illustrates the second derivative of the surface function. The two inflections in the surface correspond 
to zero-crossings in the second derivative. The bottom figure illustrates the third derivative of the 
surface function. Between the points corresponding to the two inflection points of the surface, the 
third derivative contains a zero-crossing. (From Grimson [1981b]). 
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Condition (5) implies that sgn(p) — 1. Observing that 

sgn(xy) = sgn{x) sgn(y), 

the sign of the convolved intensity function at the surface inflection points is given by 

sgn(y 2 E(xi)) = sgn(R p (p[xi))p xx {xi)). 

Condition (3) implies that/?,, does not change sign in this region of the surface and hence 

sgn(R p {p{xy))) = sgn(R,,(p(x 2 ))). 

Note that p x — 0 at xy, x 2 . The fact that there arc exactly two inflections on the surface implies that 
p x 0 over die neighborhood {xy, a^). Thus, 

sgn{p TX {x{)) 7 ^ sgifljp^flxf)), 


and tlius 

sgn(V 2 E{xy)) 7 ^ sgn{V 2 E{x 2 j). 

The second differentiability of the surface implies that there exists a point x' £ (x i} xf) such that 

V 2 /(z') = 0. | 

The contrapositive of this theorem states something important about the possible surfaces which 
can fit the known depth information. Specifically, given the conditions of the theorem, if there is a 
set of known dcpdi points to which a surface is to be fit, there cannot be two or more inflections in 
die surface between any two zero-crossing points. If diere were, there would have to be an additional 
zero-crossing there as well. 

Corollary 0.1: Suppose one is given a set of known depth points at a set of zero-crossings, along a 
developable surface. If the albedo p is non-zero and the reflectance map R is not constant for this region 
of the surface, then the surface cannot contain two or more inflection points between any pair of adjacent 
zero-crossings. % 

There are many ways of extending diis result. Some are indicated in the appendix. The most 
important extension is to consider general surfaces, rather than developable ones. We make diis 
extension in the following sections. 






4.3 The Analytic Argument 


Take a planar slice of some surface: /, normal to the image plane, along some direction (given for 
example by the angle a between the x axis and the direction of the slice). This is illustrated in Figure 
6 . At each point along tire resulting curve C, one may associate a surface orientation, or gradient, 
given by the pair of partial derivatives, p(x, y) = df{x, y)/dx, and q[x, y) = df{x, y)/dy. Thus, one 
may construct a two-dimensional space spanned by a coordinate system with axes given by p and q , 
the gradient space introduced by Huffman [1971] and used by Mackworth [1973], and by Horn [1977] 
to relate the geometry of image projections to tire radiometry of image formation. The curve obtained 
by tire planar intersection of the surface transforms into a new, parametric curve in gradient space: 


N /(0 = {pOO> 9(0. — 1} 


In fact, this curve corresponds to the mapping of the normal to tire surface, as one moves along the 
planar slice; tire subscript/ is used to indicate that this is tire normal to tire surface /. 

Because the albedo is roughly constant, the image irradianccs arc determined by tire reflectance 
map R(p, q). Thus, the irradianccs may be related to Horn’s reflectance map, in which one considers 
tire surface defined by R{p, q) in gradient space. Hence, the curve C on tire original surface will map 
onto a curve N f in gradient space and this may be projected onto the surface R(p, q) to obtain a new 
curve C'{t). At each point t along the parametric curve, tire corresponding irradiance is given by 
R{p(t), q[t)), (sealed by the constant p, which we will ignore without loss of generality). 


We are interested in the conditions on the original surface and the reflectance surface that will 
cause a zero-crossing in tire second directional derivative of the image irradiances. (By reflectance 
surface, we mean the reflectance map R(p, q ) considered as a surface in gradient space.) Let v be a 


vector in three-space with direction cosines cos a*, cos a y , and cosa* (the direction cosines refer to 
the cosine of the angle made by the vector and tire coordinate axes of tire space). Thus, a unit vector in 
the direction of v is denoted by 


v = cos a x i -f cos a y j ~F cos a z k 


where i, j, k are unit vectors along the coordinate axes. The directional derivative ofT(r), a function of 
a vector r, is the rate of change of <I> with distance s along tire direction v and is denoted by 


(v • V)<F 


<9$. <9$ , m 

cos a x - -F cos a y - -F cos 

ay dz 


dx 
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Figure 6. The Generation of Image Jrradiance. The curve C is generated by taking the intersection 
of a plane normal to the image plane with the surface. The direction of the plane is given by 
the angle a it makes with the ar-axis. By taking the surface normal at each point, this curve C 
can be mapped into gradient space, resulting in a parametric curve N f . Furthermore, this curve 
can be projected onto the reflectance map R (which defines a surface on p-q space) to obtain a 
parametric curve C along which the irradiance is given by the corresponding value of R. (From 
Grimson [1981b]). 





If the direction vector is constrained to lie in the x-y plane, then cos a z — 0 and cos a y — sin a x . 

Now consider a planar slice normal to the image along the direction v and construct the 
parametric normal curve Ny in p — q space. Then the directional derivative of irradiance along this 
slice is given by 

(v-V)£ — Rp((\ • V)p) + ^((vV)« 7 ). 

This can be rewritten as 

(v ■ V)/v = VR • (v • V)N/ 

showing that the directional derivative along the slice is given by the projection of die gradient of die 
reflectance surface R onto the tangent of the parametric p-q curve N/. Now, if there are two points 
t\, t ‘2 such that diis dot product vanishes at diosc points, then Rolle’s theorem, and the assumption 
that the surface / is twice continuously differentiable would imply die existence of a zero-crossing in 
the second directional derivative at some point between t t and t 2 . Thus, to evaluate die probability 
of a zero-crossing in some region of the image, we need to consider die probability of the vectors VR 
and (v • V)N/ being orthogonal. 

There are two cases to consider. The first is when the gradient of R vanishes at some point, cor¬ 
responding to an extremum in die reflectance surface. If we assume diat over all possible viewpoints, 
VR is uniformly distributed, dicn die probability of VR — 0 for some point (p(£), q(t)) on die 
paramedic curve Ny(f) is given by the arclcngth of the curve 



where the subscript t indicates partial differentiation with respect to die parameter t. 

The second case is when the gradient of R is non-zero and the two vectors are non-trivially 
orthogonal. Consider first die one-dimensional case in which two inflections in a developable surface 
imply the existence of a zero-crossing (as shown in Theorem 0). Since in this case q = c, the 
parametric curve N f (t) in p-q space is parallel to the p-axis, and reverses direction at each of the 
inflection points. At these points, the curvature of the parametric curve is essentially infinite, and the 
tangent can be considered to sweep through all possible directions. As a consequence, the tangent 
must be orthogonal to VR for one of those directions, and hence (v • V)E is zero at each point of 
infinite curv ature. Thus, Rolle’s theorem guarantees a zero-crossing along the slice of the image. 
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Now suppose that q c but changes slowly. In this case, the points of infinite curvature 
become points of high curvature. Here, the tangent to the parametric curve sweeps through a large 
range of directions. If we assume that the second order partial derivatives of the reflectance surface R 
arc small, then over a small region of p-q space, VR is nearly constant. Hence, in the region of the 
points of high curvature, the direction of VR is nearly constant, while the direction of the tangent to 
N/ sweeps through a large range of directions. Thus, at points of high curvature, die probability of 
(v • V)E — 0 is high. If, on the other hand, the parametric curve is locally a straight line, tire tangent 
is constant and the probability of the directional derivative vanishing is small. Thus, along a slice of 
the image, the probability that (v • V)N/ and VR arc non-trivially orthogonal, under the assumption 
that Rpp, Rpq and R qq arc small, is monotonically related to the integral of square curvature 


f (Eflfll ~ Wtt) 2 

J In? -4- n?l§ 


J ( pI + q}p 

where again the subscript t indicates partial differentiation with respect to the parameter t. 
Thus, we can combine the above discussion into the following result. 

Theorem l (Surface Consistency Theorem): Consider some region of a surface 


|/(®, y): (x, y) G G Jt C S 2 j. 


If the reflectance surface R is isotropic, its normal VR is uniformly distributed, and the second partials 
Rpp> Rpqi Rqq cire smallthen the probability of a zero-crossing in (v • for some direction v, with 
associated directional angle a, is a monotonic function of 


<%Jt 


(pt) 2 + (qtf 


and of 


f f [ ( ptqn ~ 

J hJt ( p 2 + g 2 f 


where the integral with respect to t is taken along the parametric curve obtained by slicing the image in 
some direction, and the integral with respect to is taken over all such slices of the image region S.j 

This theorem has the immediate corollary of specifying the “best” surface to fit through a set of 
known points. 




Corollary 1,1: The surface f most consistent with a set of known points provided by the visual 
system (e.g. from stereo) is that which is least likely to introduce zero-crossings not present in the primal 
sketch. By the Surface Consistency Theorem, this surface minimizes arclenglh of the parametric normal 
to the surface along each direction and curvature of the parametric normal along each direction, subject 
to passing through the known points .| 

Finally, wc would like to obtain a simple expression for this surface. Note that the curvature of a 
curve at a point is simple lire rate of change of its tangent at that point. Thus, die curve with minimum 
integral square curvature is one with a tangent as nearly constant as possible, relative to the constraints 
imposed by the known points. Next, to minimize arclenth, wc require the curve of constant tangent to 
have as small a tangent as possible. This can be done by find the curve which minimzies 

pf + qf dt 

which is equivalent to 

J ( p x cos a -f- p y sin a) 2 + (? x cos a -f q y sin a) 2 dt. 

Integrating this form over all directions a, and using the fact that p y — q x (since / is assumed to be 
twice continuously differentiable), we obtain 

fix + 2 fly + fly 

As a consequence, we obtain the following result 

Corollary 1.2: The surface f which “best” fits a set of known data in a region of the image is 
that which passes through the known points and minimizes 

/ ffixx + 2 f 2 xy+fly dxd y- 

It is interesting to note that this expression for “best” surface interpolation as the surface which 
minimizes the integral of quadratic variation has been derived elsewhere from an analysis of fitting 
this elastic plates to the known points [Grimson, 1981b, 1982]. 

Three assumptions were made in the statement of the surface consistency theorem. One was 
that tire reflectance surface R is isotropic, a second, that its normal Vi? is uniformly distributed, 
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and a third, that the second order partial derivatives of R are small. What do these assumptions 
imply about the imaging situation? 1 he assumption of an isotropic reflectance function is minor. 
The assumption of a uniform distribution of the reflectance normal is stronger. Although a stronger 
version of the surface consistency theorem can be proved by weakening this assumption, we can 
also justify the assumption by an appeal to the “general position” principle of viewer geometry and 
illuminant geometry. The notion of general position requires that all properties of the observed image 
irradianccs be roughly independent of the specific viewer position or illuminant position. In other 
words, a small movement of the viewer or the illuminant should not grossly affect the properties of 
the image irradianccs. In terms of tire reflectance surface, a slight alteration of the viewer position 
or illuminant position will generally result in a translation (and possibly a scaling) of the reflectance 
surface R relative to the p-q coordinate system. Within the proof of the theorem, the assumption 
of uniform distribution is used to argue that at any point in the image, the reflectance normal V/? 
is equally likely to be found in any direction. By applying the principle of general position, we can 
loosely argue that die illuminant is equally likely to be in any of a range of positions and diat the 
viewer is equally likely to be in any of a range of positions. As a consequence, die reflectance normal 
corresponding to a point in the image is equally likely to be found in any direction, and hence is 
uniformly distributed. The assumption of small R pp , R p(] and R^ indicates that the dieorem may not 

be valid for highly specular surfaces, but die assumption is reasonable for surfaces diat arc generally 
matte in nature. 


4.4 The Importance of the Surface Consistency Theorem 

We began tiiis paper by discussing the problem of creating complete surface representations 
from sparse data such as diat provided by stereo, and argued diat additional constraints were needed 
to effect such a surface reconstruction. The above corollary suggests diat one such constraint has 
been identified. In particular, given a class of surfaces diat pass dirough a set of known points, the 
surface which is most consistent with the imaging information is that which is least likely to introduce 
additional zero-crossings. By the surface consistency dieorem, we see diat diis surface is defined as 
the surface diat minimizes die functional of quadratic variation, or at least minimizes some measure 
which is monotonically related to this functional. In [Grimson, 1982], we combine dais fact with 
additional constraints, in order to develop a computational theory of visual surface interpolation. 


5. Examples 


T o illustrate these results, we show how inflections in the surface can cause zero-crossings in the 
convolved image, for different light source positions. Figures 7, 8, 9 and 10 show examples of a one- 
dimensional slice of a developable surface, with a Lambertian reflectance function (a perfectly diffuse 


surface which reflects flux uniformly in all directions), and the irradiance values obtained for different 


positions of a point light source. Figure 7 indicates the sample surfaces and the rough positions of 
die light sources for die different examples. Figures 8, 9 and 10 indicate sample surfaces and die 
corresponding irradiance profiles for different positions of the light source, as indicated in Figure 7. 
J he positions in die irradiance function which would give rise to a zero-crossing in die convolved 


image arc indicated. 


6. Summary 

In this paper, we have seen the need for interpolating die results of the stereo algorithm to 

create complete surface descriptions. Determining which surface to fit to the known stereo data is 

at first sight difficult, since any one of a widely varying family of surfaces could be used. We saw, 

♦ 

however, that most of diese surfaces can be discarded as possible interpolations, since they contain 
rapid fluctuations in surface orientation and these fluctuations should give rise to additional zero- 
crossings, not contained in die stereo data. Such zero-crossings would, of course, be inconsistent with 
the imaging data, and hence such a surface is considered to be unacceptable as a possible intcrpoladon 
of die known data. 1 bus, we proved diat the surface consistency constraint restricts die problem: 

The absence of zero-crossings constrains the possible surface shapes 

To make tliis constraint precise, we outlined the physics of image formation and outlined a 
derivation of Horn’s image irradiance equation. We dicn showed that this expression accounts for 
die zero-crossings of die Marr-Hildreth edge detection algoridim in die case of surface boundaries, 
surface markings, texture changes, and so forth. The equations further suggest that even in regions 
where the physical factors causing the previous types of zero-crossings are constant, there may still be 
zero-crossings, due to a change in surface shape. We used diis fact to prove the surface consistency 
theorem which relates the probability of a zero-crossing in the second derivative of an image to the 









Figure 7. Examples of One-Dimensional Surfaces. The top figure shows one surface and the 
arrows indicate the rough orientations of the light source. The numbers refer to the irradiance 
profiles in Figure 8. The bottom figure shows a second surface, with a set of rough orientations of 
the light source. The numbers refer to the profiles of Figures 9 and 10. (From Crimson [1981b]). 














Figure 8. Examples of a Surface with Two Inflections. The top figure shows a slice of a surface. 
The bottom three figures indicate irradiance profiles for different positions of the light source. 
Note that in all cases, there are six irradiance inflections. In case 3, the irradiances also undergo a 
self-shadowing, where the irradiance value is zero. The positions in the irradiance function which 
would give rise to a zero-crossing in the convolved image are indicated. (From Crimson [1981b]). 















Figure 10. Examples of a Surface with One Inflection. The top figure shows a slice of a surface. 
The bottom three figures indicate irradiance profiles for different positions of die light source. Note 
that in case 8, the irradiance profile undergoes a self-shadowing. The positions in die irradiance 
function which would give rise to a zero-crossing in the convolved image are indicated. (From 
Grimson [1981b]). 
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variation in the original surface. The surface 


consistency constraint can be used to formulate 


a com¬ 


putational theory of surface interpolation and a specific algorithm for computing the interpolated 
surface [Grimson, 1982]. 


7. Appendix 

Theorem 1 proved a relationship between any general two-dimensional surface and the prob¬ 
ability of a zero-crossing occuring in the corresponding convolved intensities. Because the theorem 
dealt with any surface, the relationship was somewhat weak. In this appendix, a set of alternate 
theorems are given which apply in cases where the surface can be considered roughly cylindrical (or 
developable). The proofs of these theorems are included for completeness. 

Theorem 2: Consider a portion of a second differentiable, developable surface oriented along the y 
axis such that f y {x, y) = q~ c, for some constant c. If the following conditions are true: 

t 

/. The surface portion contains exactly two inflection points in the x direction, at x, andxo, 

2. At the points x L and x 2 , normalized changes in albedo are dominated by normalized changes in 
reflectance, 

V 2 p ^ RpPxx 
p < R 

3. The reflectance mapR does not pass through an extremum in this region of the surface, 

4. The reflectance map R is not constant over this region of the surface, 

5. The albedo p is non-zero, 

then there exists a point x\ < x' < x 2 such that V 2 E(x') — 0. 

Proof. The signum function is defined by: 

( 1 if x > 0 
sgn(x) = < —1 if x < 0 
k 0 if rc = 0 

From the derivatives of the image equation, 

V 2 E — (V 2 p)R -f 2p x R p p x -f p(R pp pI + Rpp xx ). 






At the inflection points x\ and X 2 , p x (xi) — 0. Hence, evaluation of the equation yields 

V 2 E{xi) = V 2 p(xi)R(p(xi)) + p{xi)R p (p(xi))p xx {xi) fori = 1,2 

Condition ( 2 ) implies tliat the albedo changes are negligible in this region, so that the first term may 
be ignored, 

sgn(V 2 E(xi )) = sgn{p{xj]R p (p{x i j)p JX (x i )). 

Condition (5) implies that sgn(p) = 1 . Observing that 

sgn(xy ) = sgn{x) sgn(y), 

the sign of the convolved intensity function at the surface inflection points is given by 

sgn(V 2 E(xi)) = sgn(R p (p(xi))p xx (xi)). 

Condition (3) implies that/Z ;) does not change sign in this region of the surface and hence 

sgn{R p {p[xi))) = sgn(R p {p(x 2 ))). 

Note that p x — 0 at X{, x%. The fact that there arc exactly two inflections on die surface implies that 
p x # 0 over the neighbourhood (x h x 2 ). Thus, 

sgn(p xx (xi )) 7 ^ sgn(p xx (x 2 )), 

and thus 

sgn(V 2 E(xi)) ^ sgn(V 2 E(x 2 )). 

The second differentiability of the surface implies that there exists a point x’ £ {x h x 2 ) such that 

'V 2 E(x') = 0 ., 

Corollary 2.1: Suppose one is given a set of known depth points at a set of zero-crossings, along a 
developable surface. If the albedo p is non-zero and the reflectance mapR is not constant for this region 
of the surface, then the surface cannot contain two or more inflection points between any pair of adjacent 
zero-crossings. | 
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Theorem 3: Consider a portion of a second differentiable developable surface oriented along the y 
axis such that f y (x, y) = c, for some constant c. If the following conditions are true: 

1. The surface contains exactly one inflection point in the x direction at x u and the reflectivity R 
achieves an extremum at the point x 2 , xi xq., 

2. At the point x\ the normalized changes in albedo are dominated by normalized changes in 
reflectance of the form. 


V 2 p 

< 

RpPxx 

p 


R 


and at the point x 2 the normalized changes in albedo are dominated by normalized changes in reflectance 
of the form. 


V 2 p 

< 

RppPx 

p 


R 


3. The reflectance map R is not constant, 

4. The albedo p is non-zero, 

then there exists a point xy < x' < x 2 such that V 2 E{xf) = 0. 

Proof: As in the proof of the previous theorem, at the point x\, 

sgn(V 2 E(x x )) = sgn(R p {p{x { ))p x? .{xf)). 


At the point x 2 , R p = 0 so that 


V 2 E(x 2 ) = V 2 p{x 2 )R{p{x 2 )) + p{ x 2 )R pp (p(x 2 ))pI(x 2 ) . 

Condition (2) then implies that at this point, the normalized albedo changes are dominated by the 
normalized reflectance changes, 

sgn(y 2 E(x 2 )) — sgn(p(x 2 ))sgn(R pp (p(x 2 )))sgn(pl(x 2 )). 


Condition (4) implies that sgn(p) — 1, so that 


sgn(V 2 E(x 2 )) = sgn(R p/ ,(p(x 2 )))sgn(p 2 (x 2 )) 

= sgn(R pp (p(x 2 ))) 





There are two subcases. In the first subcase, p xx (x i) > 0. Since there is only one inflection point 
in the surface, this implies thatp^) < p(x 2 ). Then R pp (p(x 2 )) < 0 implies that 

r p{p( x 1 )) > Rpipfa)) = 0 . 

Conversely ,R pp (p(x 2 )) > 0 implies that 


^(pfci)) < r p ( p { x 2 )) = 0. 

In cither case, 

sgn(R pp (p(x 2 ))) ^ sgn{R v {p(xi))p xx {xy)). . 

in the second subcase, suppose that p xx {x x ) < 0. This implies that p(xy) > p(x 2 ). Then 
R pp (p(x 2 )) < 0 implies that 

r p{p( x 1 )) < R P {p( x 2 )) = 0 . 

Conversely, 7? p;) (p(:c2)) > 0 implies that 

-RpCpfci)) > Rp(p( x 2)) = 0. 

In cither case, 

sgn(R pp (p(x 2 ))) 7^ sgn(R p {p{x\))p xx {x y )). 

Thus, we see that 

sgn(V 2 E(x j)) 7^ sgn(V 2 E(x 2 )) 

and as before die second differentiability of the surface implies that there exists a point x' £ (xy, x 2 ) 
such that V 2 £ , (x') = 0. § 

Corollary 3.1: Suppose one is given a set of known depth points at a set of zero-crossings, along a 
developable surface. If the albedo p is non-zero and the reflectance mapR is not constant for this region 
of the surface, then if the reflectance mapR passes through an extremum, the surface cannot contain any 
inflection points between any pair of zero-crossings. | 

Theorem 4: Consider a second differentiable developable surface oriented along the y axis such 
that f y {x, y) — c, for some constant c. If the following conditions are true: 
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/. At the point x\, the surface becomes self-shadowing, that is R(x\) — 0, 

2. The reflectivity R achieves an extremum at the point x>i, X\ 7 ^ x^, 

3. At the point x 2 normalized changes in albedo are dominated by normalized changes in reflectance. 


V 2 P 


R n .Pl 

p 

< 

R ’ 


4. The reflectance rnapR is not constant over this region, 

5. Tim albedo p is non-zero, 

then there exists a point x\ < x* < x 2 such that V 2 E(x') — 0. 

Proof. The fact that the surface becomes self-shadowing implies that there is a region of the 
surface, beginning at x u such that R is constantly zero. The fact that there is an extremum in R 
for some other point implies that the intensity function must be concave down in the region of the 
extremum and concave up in the region of self-shadowing. There must be an inflection point in 
between and hence there must be a point x' such that V 2 E(x') = 0. | 

Theorem 5: Consider a second differentiable developable surface oriented dong the y axis such 
that f y {x, y ) — c, for some constant c. If the following conditions are true: 

1. At the point x\, the reflectivityR achieves an inflection point, . 

2. There exist points xq < x\ < x- 2 such that reflectance changes dominate albedo changes. 



3. The first derivative of the surface, p, is monotonic in this region, (i.e. it does not achieve an 
extremum), 

4. The reflectance mapR is not constant over this region, 

5. The albedo p is non-zero, 

then there exists a point x\ < x' < x<x such that V 2 E{x') — 0. 

Proof. The proof is very similar to the previous ones, except that in this case, by condition (2), 

sgn(y 2 E ) = sgn(p)sgn(R pp )sgn(pl) ■ 

at the points xq, X2- Then condition (4) implies that 


sgn(V 2 E ) = sgn(R pp ). 
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Condition (3) implies that 


p(zo) < p[x 1 ) < p{x 2 ) 


or 


p[x 2 ) < p[xi) < p(x 0 ) 


In either case, condition (1) dien implies that 


sgn(R pp (p(x 0 ))) jL sgn(R pp (p(x 2 ))). 


Hence, 


sgn(V 2 E(xo)) 7 ^ sgn(V 2 E(x 2 )), 


and as before, die second differentiability of die surface implies diat there exists a point x' G (xo, x 2 ) 
such that V 2 E{x') = 0. | 





8. Acknowledgements 

The author wishes to express his gratitude to David Marr, Tommy Poggio, Shimon Ullman, 
Berthold Horn, Mike Brady, Whitman Richards, Tomas Lozano-Perez, Marilyn Matz and Ellen 
Hildreth for many useful comments and discussions. 

This report describes research done at the Artificial Intelligence Laboratory of die Massachusetts 
Institute of Technology. Support for the laboratory’s artificial intelligence research is provided in 
part by die Advanced Research Projects Agency of the Department of Defense under Office of 
Naval Research constract N00014-75-C-0643 and in part by National Science Foundation Grant 79- 
23110MCS. 

9. References 

Brady, M. and Grinison, W.E.L. The perception of subjective surfaces, MIT AI Lab, Memo 582,1981. 

Brooks, M.J. Investigating the effects of planar light sources, Dept, of Computer Science, Essex 
University, Colchester, England, CSM 22,1978. 

Grinison, W.E.L. Computing shape using a theory of human stereo vision, Ph.D. Thesis, Department 
of Mathematics, Massachusetts Institute of Technology, 1980. 










REFERENCES 


34 

Grimson, W.E.L. "A computer implementation of a theory of human stereo vision," Phil. Trans. 
Royal Society of London, B. 292 (1981a), 217-253. 

Crimson, W.E.L. From Images to Surfaces: A computational study of the human early visual system , 
MIT Press, Cambridge, Mass., 1981b. 

Grimson, W.E.L. "A computational theory of visual surface interpolation," Phil. Trans. Royal Soc. 
London, B (1982), to appear. 

Helmholtz, IT. Physiological Optics, Vol. 3 , Optical Society of America, New York, 1925. 

Hildreth, E.C. Implementation of a theory of edge detection, S.M. Thesis, Department of Computer 
Science and Electrical Engineering, Massachusetts Institute of Technology, 1980. 

Horn, B.K.P. Shape from shading: a method for obtaining the shape of a smooth spaque object from 
one view, MIT Project MAC Technical Report, MAC TR-79,1970. 

Horn, B.K.P. "Obtaining shape from shading information," The Psychology of Computer Vision, P.H. 
Winston (ed), McGraw-Hill (1975), 115-155, 

Horn, B.K.P. "Understanding image irradiances," Artificial Intelligence 8 (1977), 201-231. 

Horn, B.K.P. and Bachman, B.L. Using synthetic images to register real images with surface models, 
MIT Artificial Intelligence Laboratory, Memo 437,1977. 

Horn, B.K.P. and Sjoberg, R.W. "Calculating the reflectance map," Applied Optics 18 (1979), 1770- 
1779. 

Huffman, D.A. "Impossible objects as nonsense sentences," Machine Intelligence 6, B. Meltzer andD. 
Michie(eds), Edinburgh University Press (1971), 295-393. 

Johannson, G. "Perception of motion and changing form," Scandinavian Journal of Psychology 5 
(1964), 181-208. 

Julesz, B. Foundations of Cyclopean Vision , University of Chicago Press, Chicago, 1971. 

Longuet-Iliggins, H.C. and Prazdny, K. "The interpretation of a moving retinal image," Proc. Roy. 
Soc. Lond., B 208 (1980), 358-397. 

Mackworth, A.K. "Interpreting pictures of polyhedral scenes," Artificial Intelligence A, 2 (1973), 121- 
137. 

Mayhew, J.E.W. and Frisby, J.P. "Psychophysical and computational studies towards a theory of 
human stereopsis," Artificial Intelligence (Special Issue on Computer Vision) 17 (1981). 

Marr, D. "Early processing of visual information," Philosophical Transactions of the Royal Society of 
London 275, 942 (1976), 483-534. 




35 


Marr, D. and Hildreth, E.C. "Theory of edge detection," Proc. R. Soc. LondL B 207 (1980), 187-217. 

Marr, D. and Poggio, T, "A theory of human stereo vision," Proc. R. Soc. Loud B 204 (1979), 301- 
328. 

Miles, W.R. "Movement interpretations of the silhouette of a revolving fan," Am. J. Psych. 43 (1931), 
392-505. 


Nicodemus, F.E., Richmond, J.C., Hsia, J.J., Ginsberg, I.W. and Limperis, T. Geometrical considera¬ 
tions and nomenclature for reflectance, National Bureau of Standards, NBS Monograph 160,1977. 

Silver, W. Determining shape and reflectance using multiple images, S.M. Thesis, Department of 
Electrical Engineering and Computer Science, Massachusetts Institute of Technology, 1980. 

Uilman, S. The Interpretation of Visual Motion , MIT Press, Cambridge, Mass., 1979. 

Wallach, I I. and O’Connell, I).N. "The kinetic depth effect," J. Exp. Psych. 52, 5 (1953), 571 =578. 

Wheatstone, C. Contributions to the physiology of vision. Part I. On some remarkable, and hitherto 
unobserved, phenomena of binocular vision," Phil. Trans. R. Soc. Lond 128(1838), 371-394. 

Woodham, R.J. Reflectance map techniques for analyzing surface defects in metal castings, MIT AI 
Lab, TR 457,1978. 




